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Abstract 


NASA’s first Air Traffic Management (ATM) Technology Demonstration (ATD-1) 
was created to transition the most mature ATM technologies from the laboratory 
to the National Airspace System. One selected technology is Interval Management 
(IM), which uses onboard aircraft automation to compute speeds that help the flight 
crew achieve and maintain precise spacing behind a preceding aircraft. Since ATD-1 
focuses on a near-term environment, the ATD-1 flight demonstration prototype re- 
quires radio voice communication to issue an IM clearance. Retrofit IM displays will 
enable pilots to both enter information into the IM avionics and monitor IM opera- 
tion. These displays could consist of an interface to enter data from an IM clearance 
and also an auxiliary display that presents critical information in the primary field- 
of-view. A human-in-the-loop experiment was conducted to examine usability and 
acceptability of retrofit IM displays, which flight crews found acceptable. Results 
also indicate the need for salient alerting when new speeds are generated and the 
desire to have a primary field of view display available that can display text and 
graphic trend indicators. 
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1 Introduction 


Over the next twenty years, the Federal Aviation Administration (FAA) is predict- 
ing a substantial increase in the number of revenue passenger miles flown [1], To 
accommodate the increase in traffic, the efficiency of the national airspace system 
must be increased. Interval Management (IM) is one concept that has demonstrated 
the ability to help aircraft achieve precise spacing intervals at the runway and enable 
the use of Optimized Profile Descents (OPDs) in high density airspace [2], 

IM uses on-board avionics to enable aircraft to either achieve or maintain precise 
spacing behind a target aircraft. The IM automation includes an airborne spac- 
ing algorithm that uses Automatic Dependent Surveillance-Broadcast (ADS-B) data 
from the target aircraft to compute IM commanded speeds that the pilots follow to 
precisely achieve the spacing interval. The National Aeronautics and Space Ad- 
ministration’s (NASA) Langley Research Center (LaRC) has been developing an 
implementation of IM that uses a trajectory-based spacing algorithm to compute 
commanded speeds, enabling an aircraft to perform IM operations when their target 
aircraft is on a different route. The trajectory-based IM application is one of three 
technologies selected to be demonstrated as part of the first Air Traffic Management 
(ATM) Technology Demonstration (ATD-1) [3]. 

The midterm national airspace system will not include advanced communication 
methods such as controller-pilot data link communications which could be used to 
send IM clearances and detailed information about the target aircraft’s intended 
trajectory. To compensate for the lack of controller-pilot data link communications, 
the ATD-1 concept of operations describes a midterm implementation of IM that 
uses radio communications to transmit an IM clearance from air traffic control to 
the flight deck [4]. Once pilots receive the clearance, they are expected to enter 
it into on-board avionics. Once all the necessary information is entered and all 
the constraints are met, the flight crew activates the spacing algorithm and flies 
the commanded speeds to achieve the designated spacing interval behind the target 
aircraft at the achieve-by point. Since ATD-1 focuses on a near-term environment, a 
retrofit solution is seen as the most likely implementation of IM avionics. For ATD- 
1, this retrofit solution is expected to consist of an Electronic Flight Bag (EFB) 
interface to facilitate IM clearance entry and a primary field-of-view display that 
will enable pilots to monitor the IM operation. 

One of the challenges imposed by the ATD-1 concept of operations is the communi- 
cation of complex clearances that contain an achieve-by waypoint, a spacing interval, 
and required information about the target aircraft’s route. Due to avionics communi- 
cation constraints that exist aboard certain aircraft, pilots may be required to enter 
additional information such as their own aircraft’s route, destination airport and 
descent forecast winds. To support pilots when entering the required information, 
it is necessary to ensure that the interface is designed to minimize both the time 
required to input information and the potential for errors. Likewise, it is important 
for pilots to have adequate information to monitor the spacing operation. 
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2 Background 


Previous research conducted by Eurocontrol, MITRE, and NASA investigated both 
forward-fit and retrofit IM displays. Eurocontrol investigated an IM implementation 
where IM symbology was integrated into existing cockpit displays [5-8]. In their 
final implementation the autothrottles automatically adjusted the aircraft’s speeds 
to match the IM commanded speed. The aircraft displays included IM annunciation 
on the primary flight display, the target aircraft highlighted on the navigation display, 
and an IM graphical trend indicator shown on the navigation display. The IM trend 
indicator displayed current and required spacing, spacing trend and closure rate, and 
tolerance margins. 

MITRE conducted a series of human-in-the-loop experiments to evaluate a retrofit 
IM implementation that used both an EFB interface in the forward field-of-view and 
an ADS-B Guidance Display (AGD) to present numerical data in the primary field-of- 
view [9, 10]. More recently, MITRE incorporated IM into a Gockpit Display of Traffic 
Information (GDTI) that was developed to support multiple ADS-B applications 
[ 11 ]. 

NASA investigated an IM application where IM symbology was integrated into ex- 
isting cockpit displays [12-16]. The IM symbology was located on the primary 
flight display, the navigation display, the multi-function control display unit, and 
the engine-indicating and crew-alerting system. The symbology on the primary 
flight display included the IM commanded speed displayed at the top of the speed 
tape and a green IM speed bug on the speed tape that was designed to indicate the 
difference between the IM commanded speed and the aircraft’s speed. When the 
green IM speed bug was used in conjunction with the normal aircraft speed-trend 
indicator, the pilots could determine if the aircraft was matching the deceleration 
rate predicted by the IM algorithm. Pilots were notified of new IM speed commands 
by a solid green box that appeared around the IM commanded speed for a time 
period of ten seconds. In addition to the symbology on the primary flight display, 
the IM aircraft was highlighted on the navigation display and multiple alerts and 
warnings were presented on the engine-indicating and crew-alerting system. The 
flight crews used the multi-function control display unit to auto-load information 
from IM clearances provided through controller-pilot data link communications. In 
addition, one of the human-in-the-loop experiments used an exploratory scenario to 
investigate a spacing “conformance box” that surrounded the ownship symbol on the 
navigation display, and was designed to indicate the limits of the current spacing 
error [15]. 

After NASA investigated integrated IM displays, there was considerable interest in 
designing and examining retrofit IM displays in preparation for the ATD-1 flight 
demonstration. NASA designed IM displays that consisted of an EFB interface for 
data entry and an auxiliary display in the primary field-of-view for monitoring the 
IM operation [17-19]. The EFB interface contained data entry fields, a cockpit dis- 
play of traffic information, and various pieces of IM information required to monitor 
the IM operation. The auxiliary display was similar to the numerical AGD pre- 
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sented later in this paper, and was used to present critical IM information in the 
primary held-of-view. The work described in this paper compares the previously 
used auxiliary display with a new graphical AGD to determine the usability and ac- 
ceptability of both implementations, and examines the acceptability of two different 
EFB interfaces. 


3 Experiment Design 

3.1 Experiment Procedure and Scenario Design 

3.1.1 Scenario Design 

This study investigated the acceptability and usability of retroht IM displays, while 
conducting IM operations into Phoenix Sky Harbor airport. Each scenario contained 
four IM aircraft and eighteen non-IM aircraft arriving on the EAGUL5 and KOOLY5 
arrivals to runways 25L or 26. The IM and target aircrafts’ routes always shared a 
common runway; however, they did not necessarily share a common arrival. During 
each scenario, two of the IM and target aircraft pairs began on the same arrival, 
and the remaining two pairs began on different arrivals. Within each scenario, the 
IM aircraft began approximately hve minutes prior to top-of-descent and flew until 
landing. Within this time, pilots were required to enter information about their 
aircraft’s trajectory into the IM avionics, receive an IM clearance and enter the 
appropriate data into the IM avionics, activate the IM algorithm, and fly the IM 
commanded speeds to achieve a precise spacing interval behind the target aircraft 
at the hnal approach hx. 

The scenarios were designed to simulate a midterm airspace environment where IM 
clearances were provided via radio communications and pilots manually entered the 
IM clearance information and the ownship’s intended trajectory into the IM avionics. 
To increase the realism of the scenarios, confederate air traffic controllers provided 
realistic voice communications to the IM aircraft and to the other eighteen aircraft 
in the airspace. Additionally, winds observed at Phoenix Sky Harbor Airport were 
used as the truth winds and a temporally offset wind held was used as the wind 
forecast. 

This study investigated the data entry task and the task of monitoring the IM 
operation during normal operations. As such, the scenarios within this experiment 
were scripted so that all of the aircraft were given achievable spacing clearances and 
the IM aircraft were expected to conduct uninterrupted IM operations to the hnal 
approach hx. A prior IM experiment conducted at NASA LaRG investigated various 
scenarios that required the hight crew to suspend or terminate an IM operation before 
reaching the achieve-by point [17, 18]. 



3.1.2 Experiment Procedure 


Two groups, each consisting of eight active airline pilots, participated in this experi- 
ment. Each group was present for a two day period. During this time, they received 
training and conducted eight data collection runs. The subject pilots completed 
post-run surveys, an extensive post-experiment survey, and participated in a post- 
experiment group debrief. Each group of eight pilots was split into four two-person 
crews, each consisting of a captain and first officer. Throughout the experiment the 
subject flown aircraft were rotated between four different sets of initial conditions 
and the treatment conditions were randomly ordered. To mitigate learning and fa- 
tigue effects, individual pilots within each two person crew switch roles from pilot 
flying to pilot monitoring, or vice versa, after each run. 

Comprehensive classroom sessions as well as four training scenarios were provided 
to the participants to familiarize them with the IM procedures and displays, and to 
help them acclimate to the simulators. Once the training was complete, each group 
of pilot participants flew eight data collection runs. 

Both quantitative and qualitative data were collected during each of the data col- 
lection runs. The quantitative data consisted of aircraft state data and algorithm 
performance data for each IM aircraft. The qualitative data were collected from 
electronic post-run surveys (Appendix A) completed after each data collection run, 
an extensive post-experiment survey (Appendix B), and a group post-experiment de- 
brief session. The qualitative data included measures of pilot workload and usability 
and acceptability ratings of the IM displays. 


3.1.3 Participants 

The pilots who participated in this study were all active airline pilots with glass 
cockpit experience. The pilots’ ages ranged from 44 to 65 years, and they had an 
average of 13,600 hours of commercial flight experience. In order to ensure that the 
pilots were able to use their normal crew procedures, each two person flight crew 
consisted of a captain and first officer from the same airline. 

3.2 Pilot Tasks 

Within this study, there were three major tasks related to the IM operation: entering 
the IM aircraft’s trajectory intent information into the IM avionics, receiving an IM 
clearance and entering the necessary information into the IM avionics, and flying the 
speeds commanded by the IM avionics to achieve a precise spacing interval at the 
final approach fix. 

During cruise, the flight crews were responsible for entering information about their 
aircraft’s intended trajectory into the IM avionics. This information was required to 
be manually entered because it was assumed that a limited amount of the information 
from the flight management system would be available to the retrofit IM application. 


9 



As a result, pilots entered their aircraft’s cruise speed, cruise altitude, the speed 
at which they were expected to transition from Mach to calibrated airspeed, their 
destination airport, and their aircraft’s route. Additionally, pilots were required 
to manually enter their aircraft’s descent forecast winds. Both of the data entry 
interfaces examined in this experiment contained several selectable elements to help 
streamline the data entry process. 

Air traffic control provided the flight crews with an IM clearance before the aircraft 
crossed 30,000 ft. If the target and IM aircraft were on the same arrival, air traffic 
control issued a "when able" clearance, instructing the flight crews to activate the 
interval management clearance as soon as they entered the necessary information. 
Alternatively, when the IM and target aircraft were on different arrivals, air traffic 
control provided the flight crew with a clearance instructing them to activate the 
IM operation when they entered the Terminal Radar Approach Control (TRACON) 
airspace. After receiving the IM clearance, the pilot monitoring was expected to 
enter the spacing goal, target aircraft ID, and the target aircraft’s route into the 
EFB interface. After the data was entered and the pilot flying had crosschecked the 
information, the flight crew activated the IM avionics at the proper time; either at 
the designated waypoint if the IM and target aircraft were on different routes or 
immediately if the IM And target aircraft were on the same route. Once IM was 
activated and the IM avionics had all the necessary information to compute valid 
speed guidance, a commanded speed was displayed on the primary field-of-view 
display. 

When IM was activated and speed guidance was displayed, the flight crews were 
required to verify that the speed was safe to fly and then enter it into the Mode 
Control Panel (MCP) speed window. As the commanded speed changed, pilots 
were required to verify that each new commanded speed was safe to fly and update 
the speed in the MCP speed window in addition to carrying out their normal tasks. 
When the IM aircraft reached the final approach fix, the pilots slowed to their landing 
speed and prepared for landing. If the flight crew did not think the speed was safe 
to fly or they were not comfortable flying it at any point during the flight, they had 
the option of contacting air traffic control and canceling the IM operation. 

3.3 Data Entry Interface 

This experiment examined two different EFB interfaces that pilots used to enter 
information into the IM avionics. The first interface, which will be referred to as 
the menu-entry EFB interface, contained separate pages for each new piece of data 
that had to be entered. The second interface, which will be referred to as the 
multi-entry EFB interface, enabled pilots to enter multiple pieces of data on a single 
page. Since it was assumed that a limited amount of the information from the flight 
management system would be available to the retrofit IM application, pilots were 
required to enter ownship trajectory intent information and the descent forecast 
winds in addition to the IM clearance (table 2). It is likely that this is a worst-case 
data entry scenario. 
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The ownship trajectory intent data that pilots were required to enter were the IM 
aircraft’s cruise altitude, cruise Mach, the transition between Calibrated Airspeed 
(CAS) and Mach, the IM aircraft’s destination airport, and the IM aircraft’s route. 
In addition to the airport, pilots were required to enter the altitude, speed, and 
direction for a set of four descent forecast winds. Lastly, pilots were required to 
enter the following data from the spacing clearance: the spacing goal assigned by 
air traffic control, the target aircraft’s Identihcation (ID), and the target aircraft’s 
intended route. The achieve-by waypoint and terminate waypoint were automatically 
set as the hnal approach hx to conform with the ATD-1 concept of operations [4]; 
however, the pilots could override the pre-populated value if desired. 

The main CDTI page of both EFB interfaces contained certain symbology designed 
to improve pilots’ situational awareness of the IM operation (hgures 1 and 5). First, 
the target aircraft was highlighted in green on the CDTI and the data block of the 
target aircraft was expanded to include the target aircraft’s ID. Secondly, there were 
two cyan boxes added to the lower portion of the main CDTI pages. The top box 
contained text that indicated the IM mode and the target aircraft’s ID. The IM mode 
indicator displayed modes that indicated the spacing algorithm was either actively 
spacing, suspended, unable to compute a speed, or waiting for various constraints 
to be met before providing a commanded speed. The mode appeared as green text 
when in spacing mode and as white text when in any other mode. The bottom cyan 
box displayed IM status messages when the IM or target aircraft were not on the 
expected route, if the target aircraft’s ADS-B signal was lost, or if there was another 
problem that prevented the IM algorithm from providing speed guidance. 


Table 2: Data Pilots were required to enter 


Ownship Data: 

• Cruise Altitude 

• Cruise Mach 

• Descent Mach/CAS 

• Descent forecast winds 

• Destination airport 

• Ownship route^ 

— Arrival^ 

— Transition^ 

— Approach^ 

— Approach Transition^ 


Spacing Clearance Data: 

• Achieve-by waypoint^ 

• Terminate waypoint^ 

• Spacing goal 

• Target aircraft ID^ 

• Target aircraft route^ 

— Arrival^ 

— Transition^ 

— Approach^ 

— Approach Transition^ 


^Pilots were able to select these items from a list of options instead of manually typing the 
information. 

■^In the menu-entry EFB interface, the target aircraft’s ID could be selected from a list of 
options if the target aircraft was within ADS-B range when the pilot was entering the IM clearance 
information. Otherwise the target aircraft’s ID had to be manually typed in. 

^These fields were automatically populated with the final approach hx. Pilots could select a 
different waypoint from a list of options if desired, but they were not required to do so for this 
experiment. 
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Figure 1: Menu-entry EFB display 
CDTI page 


Figure 2: Menu-entry EFB display 
data input menu 


3.3.1 Menu-Entry 

The menu-entry EFB interface is roughly based off of the EFB interface used in a 
previous human-in-the loop experiment [17,18]. Since ownship trajectory data may 
not be available from the flight management system of all aircraft, modifications were 
made to the interface that enabled pilots to directly enter their aircraft’s trajectory 
intent information into the IM avionics. Additional modifications included the re- 
moval of the scheduled time of arrival functionality and the addition of the terminate 
waypoint field to improve conformance with the minimum operational performance 
requirements that are in the process of being developed. The revised EFB interface 
required pilots to manually enter several additional pieces of information, increasing 
the workload associated with the data entry task. 

The menu-entry EFB interface contains two menus that the pilot can select from the 
main page (figure 1). Pressing the OWNSHIP INFO button activates a menu that 
shows all of the data entry fields for owship data (figure 2). Similarly, pressing the 
SPACING CLEARANCE button shows all of the data entry fields for IM clearance 
data. These two menus were designed to prevent confusion by separating the ownship 
data from the IM clearance data. Pressing any of the data entry fields switched the 
interface to a data entry page, which was either a text entry page that allowed pilots 
to manually type in information or a page that allowed pilots to select the desired 
parameter from a list of options. Figure 4 is an example of a text input page and 
figure 3 is an example of the route selection page. 

When all necessary data were entered, the ACTIVATE button on the bottom right 
of the main CDTI page switched from inactive to active. At that time, the other 
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Figure 3: Menu-entry EFB display 
CDTI page 


Figure 4: Menu-entry EFB display 
data input menu 


pilot could easily cross-check the information that was entered by selecting either 
the ownship data menu or the spacing clearance menu. Once all of the information 
was crosschecked, pilots pressed the ACTIVATE button to activate the IM algo- 
rithm. 


3.3.2 Multi-Entry 

The difference between the multi-entry and menu-entry EFB pages was that the 
multi-entry interface allowed multiple pieces of data to be entered on a single page, as 
opposed to a single entry on each page on the menu-entry EFB pages. To simplify the 
interface design most of the selection pages were identical to those used in the menu- 
entry interface. To enter data into the IM avionics, pilots pressed the INTERVAL 
SPACING button on the main CDTI page (figure 5), bringing them to a series of 
data entry pages. 

The top half of each data entry page contained multiple data entry fields, and the 
bottom half of each page contained a keyboard (figure 6). When pilots were entering 
text data, they pressed one of the data entry fields, causing a cursor to appear in 
the data entry field and enabling the pilots to type in the appropriate data. To 
enter selectable data, the pilots pressed the data selection button, which brought 
them to the appropriate data selection page. After the information was selected, the 
interface returned to the main data entry page. The data selection pages used in 
the multi-entry EFB interface were identical to the data selection pages used in the 
menu-entry EFB interface. 
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Figure 5: Multi-entry EFB display 
CDTI page 


Figure 6: Multi-entry EFB display 
data input page 


When pilots had finished entering data into a particular page, they could move to 
the next data entry page by pressing the PAGE button. At any time in the data 
entry process, the pilot could press the EXIT button to clear all of the data they had 
entered and return to the main CDTI page. The pilots could also press the SAVE & 
EXIT button to save all of the data they had entered and return to the main CDTI 
page. Once all required information was entered, the ACTIVATE button became 
selectable, enabling pilots to activate the IM operation. 

3.4 Primary Field-Of-View Displays 

Two primary field-of-view displays were investigated in this experiment. The first 
display, which will be referred to as the numerical AGD, is similar to a display 
that has been used in several IM experiments and flight tests [9, 10, 17-20]. The 
numerical AGD contained three helds that could only display numbers and a green 
Light-Emitting Diode (LED) indicator (figure 7). The second display, which will 
be referred to as the graphical AGD, contained two graphical indicators and text 
information that were unable to be displayed on the numerical AGD (hgure 8). 


3.4.1 Numerical AGD 


The numerical AGD displayed three values, 
which are shown in figure 7: the IM commanded 
speed (feature 1), a fast /slow indicator (feature 
2), and the spacing error (feature 3). The far 
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Figure 7: Numerical AGD 
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left indicator displayed the IM commanded speed that pilots were expected to dial 
into the aircraft’s MCP speed window to close the control loop. When the IM com- 
manded speed changed, the LED light in the top left corner of the nnmerical AGD 
illnminated nntil the new commanded speed was dialed into the MCP speed window 
(featnre 4). The middle indicator was a fast/slow indicator that showed pilots the 
difference between the IM aircraft’s speed and the speed commanded by the spacing 
algorithm. A positive valne indicated that the aircraft was too fast and a negative 
valne indicated that the aircraft was too slow relative to the IM commanded speed. 
The held on the far right showed the predicted spacing error at the hnal approach 
hx. The spacing error was compnted by determining the difference between both air- 
crafts’ estimated times of arrival, assnming that they wonld hy the pnblished speeds 
to the achieve- by point, and snbtr acting the spacing interval assigned by air traffic 
control. A positive spacing error indicated that the IM aircraft was projected to have 
a larger spacing interval at the achieve-by point than the assigned spacing goal (a 
late arrival), and a negative spacing error indicated that the aircraft was projected 
to have a smaller spacing interval at the achieve-by point (an early arrival). 


3.4.2 Graphical AGD 


The graphical AGD was a three inch by three 
inch glass display that enabled graphical trend 
indicators and text data to be presented in the 
primary held-of-view. The graphical AGD con- 
tained a fast/slow indicator, the IM commanded 
speed, the IM mode, the target aircraft’s ID, 
an IM progress indicator, and a space for sta- 
tns messages (hgnre 8). 
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Fignre 8: Graphical AGD 


A graphical fast / slow indicator was displayed on 
the leftmost section of the graphical AGD (fea- 
tnre 1). The fast /slow indicator consisted of 
three white diamonds spaced 20 knots apart and 
a sqnare green bng, and showed the aircraft’s 
speed deviation from the IM commanded speed. 

To snpport the pilot task of maintaining a speed 

within ten knots of the IM commanded speed, the nnmeric speed deviation was dis- 
played inside the green box when the speed deviation was greater than ten knots. 
An acceleration arrow was added to the fast /slow indicator to enable the pilots to 
determine whether they were trending toward the commanded speed. The tip of the 
acceleration arrow showed the projected speed deviation in ten seconds if the air- 
craft’s acceleration and the acceleration predicted by the spacing algorithm remained 
the same. 


The middle colnmn of the graphical AGD contained the IM commanded speed (fea- 
tnre 2), an IM mode indicator (featnre 3), and the target aircraft’s Id (featnre 4). 
The IM speed was displayed in large green nnmbers, which wonld be shown in re- 
verse video when the IM commanded speed changed (black characters on a green 
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background), notifying pilots to dial the new IM speed into the MCP speed window. 
Once the pilots dialed the IM speed into the speed window, the IM speed indicator 
would return to its normal state. The IM mode indicator displayed modes that in- 
dicated that the spacing algorithm was either actively spacing, suspended, unable 
to compute a speed, or waiting for various constraints to be met before providing a 
commanded speed to the flight crew. The mode appeared as green text when the 
aircraft was actively spacing and as white text when in any other mode. Whenever 
the mode changed, a solid green box appeared around the mode indicator for a time 
period of ten seconds. The target aircraft’s ID was displayed below the IM mode 
indicator, and was shown using text that matched the color of the IM mode. 

The rightmost column of the graphical AGD contained an IM progress indicator. 
This indicator was intended to provide pilots with extra situational awareness of 
the IM operation by showing them the IM aircraft’s spacing error in relation to the 
IM feasibility bounds. The indicator was displayed using a long vertical white line 
with tick marks at the top, middle, and bottom. The top and bottom ends of the 
progress indicator showed the IM feasibility bounds, which were approximated as 
±10 percent of the IM aircraft’s time-to-go. A green triangle indicated the spacing 
error in relation to the IM feasibility bounds. The spacing error was computed by 
determining the difference between both aircrafts’ estimated times of arrival with 
the assumption that they would fly the published speeds to the achieve- by point, 
and subtracting the spacing interval assigned by air traffic control. The assigned 
spacing interval was no longer reachable when the green triangle was at the top or 
bottom of the indicator, informing the flight crew that the spacing algorithm did 
not have enough control authority to achieve the assigned spacing goal. When this 
occurred, a message was displayed on the graphical AGD directing the flight crews 
attention to the EFB interface, which displayed a message that explained that the 
spacing goal was no longer achievable. 

The last pieces of information displayed on the graphical AGD were two status 
messages shown in the blank space below the other symbology (feature 6). There 
were two status messages that could be displayed: DRAG REQ and EFB MSG. The 
DRAG REQ message was displayed when the aircraft’s speed was more than ten 
knots higher than and diverging from the IM commanded speed. The EFB MSG 
symbology was used to direct the pilot’s attention to a larger message box on the 
EFB display, where more detailed messages were displayed. 

3.5 Facilities 
3.5.1 Simulators 

This experiment was conducted in the Air Traffic Operations Laboratory at NASA 
LaRG. This facility contains a number of desktop aircraft simulators, air traffic 
control stations, and pseudo pilot stations. Each of the simulation platforms can be 
used together to simulate complex air traffic operations. 
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Figure 9: Dual crew desktop simulator displays 


All of the pilots who participated in this experiment flew dual crew desktop sim- 
ulators that emulated a large transport aircraft. Each simulator contained a high 
fidelity six degree of freedom dynamics model and aircraft displays shown on three 27 
inch touchscreen monitors (figure 9). Pilots interacted with the desktop simulators 
through either a mouse or touchscreen interface. Each simulator contained displays 
and systems found on large transport aircraft, including right and left primary flight 
displays, right and left navigation displays, right and left EFBs, a flight management 
system, auto-throttles, radios, and ADS-B in/out. In addition to normal aircraft sys- 
tems, retrofit IM avionics were emulated by the desktop simulators. New pages that 
were added to the EFB interface enabled pilots to enter information into the IM 
avionics. Information from the IM avionics was displayed on the EFBs and on two 
auxiliary IM displays located above the right and left primary flight displays. 


Spacing Algorithm 

The spacing algorithm used in this experiment was a trajectory based algorithm 
developed at NASA LaRC, known as the eleventh version of the Airborne Spacing for 
Terminal Arrival Routes (ASTAR-11) spacing algorithm [21]. The spacing algorithm 
used speeds from the published standard terminal arrival route along with the IM 
and target aircrafts’ intended routes to compute their estimated times of arrival at a 
designated point in space, referred to as the achieve- by point. The estimated times of 
arrival for the IM and target aircraft were used in conjunction with the spacing goal 
to calculate the spacing error. Once the spacing error was calculated, proportional 
control was used to compute the amount of speed compensation that the IM aircraft 
needed to null the spacing error by the achieve- by point. The speed compensation 
was added to the published speeds, quantized into discrete increments, and displayed 
to the flight crew. Since this study investigated a retrofit IM implementation, the 
pilots were required to close the control loop by entering the IM commanded speeds 
into the MCP speed window. 

3.6 Experiment Design and Independent Variables 

Each scenario in this experiment was split into two flight segments: entering the 
IM clearance and monitoring the IM operation. To enable the investigation of EFB 
displays and primary field-of-view displays, these two segments of flight were assumed 
to be independent from each other, and the performance metrics were carefully 
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selected to pertain to only one segment of flight. Each gronp of pilots condncted two 
replicates of fonr distinct scenarios — one with the captain flying and the other with 
the first officer flying (tables 3 and 4). 

Table 3: Experiment design dnring data entry phase of flight 


Out of ADS-B Range 
In ADS-B Range 


EFB Displays 


Menu-entry Multi-entry 


First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 


Table 4: Experiment design when monitoring IM 


Nominal 

Perturbation 


Primary Field-Of-View Displays 
Numerical AGD Graphical AGD 


First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 

First Officer Flying 
Gaptain Flying 


The independent variables associated with the data entry phase of flight were the 
previonsly discnssed EFB displays and whether the target’s ADS-B information was 
available when IM was initiated (table 3). Dnring each scenario two ont of the fonr 
target aircraft began on a different arrival than the IM aircraft and were ont of 
ADS-B range when IM operations began. The remaining two target aircraft began 
on the same ronte as the IM aircraft and thns were within ADS-B range when the 
IM operation began. On the menn-entry interface, pilots were able to select the 
target aircraft’s ID from a list when it was in ADS-B range. If the target was not 
in ADS-B range when the IM clearance was entered, pilots were reqnired to type in 
the target aircraft’s ID. On the mnlti-entry interface, pilots were always reqnired to 
mannally type in the target aircraft’s ID regardless of whether the target aircraft 
was in ADS-B range. 

The independent variables associated with condncting IM operations were the pre- 
vionsly described primary field-of-view displays and the target aircraft’s deviation 
from its expected speeds (table 4). Dnring the nominal cases the target aircraft 
flew their pnblished speeds, which matched the speeds that the spacing algorithm 
predicted. Dnring the pertnrbation cases, the target aircraft was slowed below the 
pnblished speeds between TRACON entry and the start of final approach. This 
cansed the spacing error on the graphical AGD progress indicator to increase nntil it 
was close to the early feasibility bonnd and the spacing error on the nnmerical AGD 
to increase in valne. 
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3.7 Experiment Limitation 


A limitation of this study that the reader should be aware of is that this study 
was performed using desktop computer simulators instead of a flight simulator cab. 
While this does not reduce the importance of the findings, this limitation should be 
taken into account when interpreting the data. Particularly, certain metrics, such 
as the pilot’s reaction time to new IM commanded speeds, should only be used for 
comparison purposes within this experiment. 


4 Results: Data Entry Phase of Flight 

4.1 Pilot Workload 

The pilot participants used the Modified Cooper-Harper (MCH) workload scale to 
rate their workload level when entering IM information into the EFB interfaces. 
The MCH workload scale contains a flowchart that the pilot participants followed to 
determine their workload level [22]. A workload level of one indicated that the task 
was very easy and a workload level of ten indicated that the task was impossible. 
For the purposes of this study, ratings of one to three were considered acceptable 
responses. Overall, the mean pilot workload rating when using the menu-entry FFB 
interface was 2.0 {SD—0.9, N—64), and the mean workload rating when using the 
multi-entry EFB interface was 2.3 {SD—I. 4 , N—64)- The pilot workload ratings were 
less than three on the MCH rating scale {p<0.0005), indicating that the workload 
was acceptable. A series of Wilcoxon signed rank tests with Bonferroni adjustment 
revealed no statistically significant interaction effects between the EFB interfaces and 
whether the target aircraft ADS-B information was available when IM information 
was entered. Additionally, there was no difference between the mean workload when 
pilots used the menu-entry EFB and the multi-entry EFB interface {p— 0.074), or 
the workload for aircraft that began within or out of ADS-B range of their target 
aircraft {p— 0.697). 

4.2 Acceptability of Data Entry Time 

One metric of considerable importance was the acceptability of the time it took 
pilots to enter all of the data into the EFB interface. The data entry task examined 
in this experiment assumed limited connectivity between the IM aircraft’s flight 
management system and the IM avionics where pertinent information is not available 
to the IM application, and is seen as a worst-case scenario. One goal of this study 
was to determine whether the worst-case data entry task was acceptable to pilots 
for the ATD-1 flight demonstration or whether increased connectivity between the 
flight management system and IM avionics was required. 

Pilots were asked to rate whether the amount of heads-down time required to enter 
information into the EFB interface was acceptable using a seven-point scale ranging 
from one (completely agree) to seven (completely disagree) (figure 10). The mean 
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Completely Agree 
Moderately Agree 
Slightly Agree 
Neutral 
Slightly Disagree 
Moderately Disagree 
Completely Disagree 

In ADS-B Range: No Yes No Yes 

EFB Interface: Menu-Entry Multi-Entry 




No Yes No Yes 
Menu-Entry Multi-Entry 



Figure 10: Pilot ratings de- 
scribing whether they agreed 
that the amount of data entry 
was acceptable 


Figure 11: Pilot rat- 

ings describing whether 
they agreed that the 
EFB interface was intu- 
itive 


Figure 12: Pilot rat- 

ings describing whether 
they agreed that the 
EFB interface was ac- 
ceptable 


rating of the menu-entry EFB interface was 6.1 {SD—1.2, N—64), and the mean 
rating for the multi-entry EFB interface was 5.9 {SD—1.5, N—64)- Overall, the 
mean acceptability ratings were statistically signihcantly higher than 4.5 {p<0.0005), 
indicating that the amount of heads down time needed to input the data into the EFB 
was acceptable. A series of Wilcoxon signed rank tests with Bonferroni adjustment 
revealed no interaction effects between the EFB interfaces and whether the target 
aircraft ADS-B information was available when IM information was entered. In 
addition, there was no difference between the mean acceptability of the menu-entry 
and multi-entry EFB interfaces (p<0.315), or the acceptability for aircraft that 
began within or out of ADS-B range of the target aircraft {p<0.880) 


4.3 Intuitiveness 

Pilots were also asked to rate the intuitiveness of the EFB interfaces using a seven- 
point scale ranging from one (completely disagree) to seven (completely agree) that 
the IM EFB interfaces were intuitive (hgure 11). The mean intuitiveness rating of 
the menu-entry EFB interface was 6.3 {SD—1.0, N—64), the mean intuitiveness 
rating for the multi-entry EFB interface was 5.8 {SD=1.6, N—64)- For each of the 
four treatment combinations, the mean intuitiveness ratings were statistically sig- 
nihcantly higher than 4.5 {p<0-009), indicating that pilots found entering the IM 
clearance information into the EFB to be easy and intuitive. A series of Wilcoxon 
signed rank tests with Bonferroni adjustment revealed a statistically signihcant dif- 
ference between the pilot intuitiveness ratings for the menu-entry and multi-entry 
EFBs when the target aircraft was within ADS-B range. Pilots rated the menu- 
entry EFB as more intuitive than the multi-entry EFB. There were no statistically 
signihcant differences between any of the other treatment combinations. 
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4.4 Frustration Associated with the EFB Interfaces 


Pilots were also asked to rate their frustration associated with the use of the EFB 
interfaces using a seven-point scale ranging from one (completely disagree) to seven 
(completely agree) that entering information into the EFB resulted in a minimal 
amount of frustration (figure 12). The mean frustration rating of the menu-entry 
EFB interface was 6.3 {SD—1.0, N—64), and the mean acceptability rating for the 
multi-entry EFB interface was 5.8 {SD—1.6, N—64)- For each of the four treat- 
ment combinations, the mean ratings were significantly higher than 4.5 {p<0.005), 
indicating that pilots found entering the IM clearance information into the EFB to 
result in a minimal amount of frustration. A series of Wilcoxon signed rank tests 
with Bonferroni adjustment revealed a statistically significant difference between 
the data entry ratings for the menu-entry and multi-entry EFB interfaces when the 
target aircraft was within ADS-B range. Pilots rated the menu-entry EFB as less 
frustrating {M—6.5, SD—0.7) than the multi-entry EFB {M—5.5, SD—1.9). There 
were no statistically significant differences between any of the other treatment com- 
binations. 

4.5 General Pilot Comments 

Comments from the post-experiment survey and the post-experiment debrief re- 
vealed that a majority of pilots thought that the data entry required would be 
acceptable, but not ideal. Overall, 15 out of the 16 pilot participants rated the 
menu-entry EFB interface as minimally acceptable and 14 out of the 16 pilot par- 
ticipants rated the multi-entry interface as minimally acceptable.^ One of the pilots 
rated both EFB interfaces as unacceptable and desired an interface that merged the 
ability to enter multiple pieces of information on a single page with a more intuitive 
layout. He also thought that the multi-entry interface should enable pilots to type in- 
formation into a scratch-pad and then select the appropriate data-entry field instead 
of selecting the appropriate data entry field and then typing in the data, increasing 
conformance with current multi-function control units. Comments provided in the 
post-experiment debrief indicated that both interfaces were minimally acceptable, 
but pilots highly preferred the menu-entry interface over the multi-entry interface. 
The pilot participants stated that they felt that menu-entry interface was a lot more 
intuitive and acceptable than the multi-entry interface. The reasons for this included 
less intuitive page navigation on the multi-entry EFB interface and wind entry fields 
mismatched with the descent forecast winds displayed on the aircraft’s multi func- 
tion control unit. Lastly, the pilots commented that they thought the amount of 
data they were required to enter was very high. To reduce workload and increase 
acceptability when entering ownship data, it is recommended that future EFB in- 
terfaces enable pilots to auto-load their descent forecast winds. Other methods of 
decreasing the amount of data entry, such as increasing communication between the 
flight management system and the IM avionics, could also be considered. 

■^Pilots were asked about minimal acceptability to determine whether the proposed displays and 
amount of data they were required to enter would be acceptable for the ATD-1 flight demonstration. 
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There was also a less than ideal interaction between the interface and procedures 
that was observed throughout the simulation. Within this simulation, pilots were 
provided two types of IM clearances. “When able” IM clearances were provided when 
the IM and target aircraft were on the same route, and “at” clearances were provided 
when the IM and target aircraft were on different routes, instructing them to activate 
the IM equipment when their aircraft crossed the meter hx (when they entered the 
TRACON). When pilots received an “at” clearance, they were instructed to draw a 
circle around the activation waypoint on their navigation display as a reminder to 
activate the IM avionics when they crossed the activation waypoint. Despite all of 
the training provided, several pilots attempted to override the prepopulated achieve- 
by or terminate waypoints with the activation waypoint. Often times, this behavior 
was caught by the other pilot and corrected; nevertheless, this behavior continued 
throughout the experiment. To solve this problem, the procedures could be modihed 
or a new data held for the activation waypoint could be added to the interface. 


5 Results: Monitoring Phase of Flight 

5.1 Pilot Acceptability and Workload 

The primary data collected during the monitoring phase of hight were pilot ratings 
of the usefulness and acceptability of the displays. The objective of these ratings was 
to evaluate the acceptability of the graphical and numerical AGDs, and determine if 
one was more acceptable than the other. To examine the acceptability and usability 
of the displays, pilots were asked a series of questions in the post-run and post- 
experiment surveys. Additional information was gathered during a post-experiment 
debrief. 


5.1.1 Pilot Workload and Situational Awareness 

The pilot participants used the MCH workload scale to rate their workload level 
when conducting IM operations into Phoenix Sky Harbor Airport. For the pur- 
poses of this study, ratings of one to three were considered acceptable responses. 
Overall, the mean pilot workload rating when using the numerical AGD was 2.6 
{SD—I. 4 , N— 64 ), and the mean workload rating when using the graphical AGD 
was 1.9 {SD—1.5, N—64)- During the monitoring phase of flight, the mean workload 
ratings were 3 or less on the MGH rating scale (p < 0.015), indicating that the 
workload was acceptable. A series of Wilcoxon signed rank tests with Bonferroni ad- 
justment revealed no statistically signihcant interaction effects between the primary 
held-of-view displays and the target aircraft’s deviation from its expected speeds. 
Although the mean workload rating was higher when using the numerical AGD than 
the graphical AGD, the difference was less than one unit on the MGH rating scale 
(p = 0.010), indicating the display used had a relatively small operational impact 
on workload. 
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Completely Agree 
Moderately Agree 
Slightly Agree 
Neutral 
Slightly Disagree 
Moderately Disagree 
Completely Disagree 

Perturbation: No Yes No Yes 

Display: Numerical Graphical 





No Yes No Yes 

Numerical Graphical 


Figure 13: Pilot ratings de- 
scribing whether they agreed 
that the IM displays provided 
adequate situational aware- 
ness 


Figure 14: Pilot rat- 

ings describing whether 
they agreed that the IM 
displays were intuitive 


Figure 15: Pilot rat- 

ings describing whether 
they agreed that the 
IM commanded speeds 
made sense to them 


The pilot participants were also asked to rate whether they agreed that the IM 
displays provided adequate situational awareness using a seven point Likert scale 
ranging from one (completely disagree) to seven (completely agree) that the displays 
provided adequate situational awareness (hgure 13). The mean rating provided when 
using the numerical AGD was 5.0 {SD^l.9, N—64), and the mean rating provided 
when using the graphical AGD was 6.1 {SD—1.5, N—64)- A series of Wilcoxon 
signed rank tests with Bonferroni adjustments revealed a statistically signihcant 
difference between the mean situational awareness ratings for the graphical AGD 
when a perturbation occurred and the numerical AGD without a perturbation. For 
the treatment combinations with the numerical AGD with a perturbation, and the 
graphical AGD both with and without a perturbation, the mean ratings were 4.5 or 
higher, indicating that the primary held-of-view displays provided adequate situa- 
tional awareness. However, for the scenario with the AGD without a perturbation, 
the mean rating was not statistically signihcantly higher than 4.5. 


5.1.2 Intuitiveness of Displays 

Pilots were also asked to rate the intuitiveness of the displays using a seven-point Lik- 
ert scale ranging from one (completely agree) to seven (completely disagree) that the 
IM displays were intuitive (hgure 14). The mean intuitiveness rating of the scenarios 
that used the numerical AGD was 4.9 {SD^l. 8 , N—64), the mean intuitiveness 
rating for the scenarios that used the graphical AGD was 6.2 {SD—I. 4 , N—64)- For 
all four treatment combinations, the mean intuitiveness ratings were 4.5 or higher, 
indicating that the primary held of view displays were intuitive. A series of Wilcoxon 
signed rank tests with Bonferroni adjustment revealed that the mean intuitiveness 
ratings for the graphical AGD when a perturbation occurred was signihcantly higher 
than the numerical AGD both with and without a perturbation. 
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5.1.3 Understanding of IM commanded speeds 


When creating this study, it was hypothesized that the graphical IM progress indica- 
tor may increase the pilots’ understanding of the commanded speeds. For instance, 
when the progress indicator indicated the aircraft was projected to arrive early, the 
IM aircraft should be commanded a speed below the nominal prohle. Conversely, 
when the progress indicator showed the aircraft was projected to arrive late, the 
IM aircraft should be commanded a speed above the nominal prohle. Additionally, 
when the spacing error was returning to zero, the commanded speed should trend 
toward the nominal speed prohle. It should be noted that these behaviors are highly 
dependent on the particular spacing algorithm used in this experiment, and should 
not be generalized to all spacing algorithms. 

Pilots were asked to use a seven point scale to rate whether the IM commanded 
speeds made sense to them (hgure 15). A rating of seven indicated that the pilots 
completely agreed that the IM speeds made sense to them and a rating of one 
indicated that the pilots completely disagreed. The mean rating provided when 
using the numerical AGD was 5.9 {SD—1.3, N—64), and the mean rating provided 
when using the graphical AGD was 6.3 {SD—1.1, N—64)- ^ series of Wilcoxon signed 
rank tests with Bonferroni adjustment revealed no statistically signihcant differences 
in the mean pilot understanding of IM speeds between the numerical AGD and the 
graphical AGD (p = 0.966), or between the perturbation and nominal scenarios 
(p = 0.601). This suggests that the graphical IM progress indicator (the early/late 
indicator) did not increase the pilots’ understanding of the commanded speeds. Pilot 
comments about the IM progress indicator, which are discussed in the next sections, 
conhrmed that they did not use the IM progress indicator often and that many pilots 
did not hnd the information very useful. 


5.1.4 Usefulness of Symbology on Displays 

In the post-experiment survey, pilots were asked to rate the usefulness of display 
elements on both the numerical AGD and the graphical AGD (hgure 16). A majority 
of pilots rated the IM commanded speed indication, the IM mode indicator, and the 
target’s ID on the graphical AGD as very useful or required for IM. A majority of 
pilots also rated the usefulness of the fast/slow indicator, the acceleration arrow on 
the fast/slow indicator, and the IM progress indicator as slightly useful to very useful. 
The ratings of interface elements on the numerical AGD showed a similar trend. A 
majority of pilots rated the IM commanded speed as very useful to required for IM, 
and a majority of pilots rated the fast/slow indicator on the numerical AGD as not 
useful at all to moderately useful. The usefulness ratings of the IM commanded 
speed were lower for the numerical AGD than for the graphical AGD. It is suspected 
that this difference was caused by a lack of saliency of speed change alerting on the 
numerical AGD. 

With the exception of the IM progress indicator, pilot comments during the post- 
experiment debrief matched the survey data presented in hgure 16. A majority of 
pilots stated that they found the fast/slow indicator along with the acceleration 
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Figure 16: Usefulness ratings of primary field-of-view display symbology 


arrow somewhat useful. Despite rating the IM progress indicator as moderately 
useful, pilots in the post-experiment debrief unanimously stated that they had not 
used it and did not believe it should be a requirement in future displays. Additionally, 
none of the pilots had not noticed the large increase in spacing error that occurred 
during perturbation scenarios, suggesting that they were not consistently monitoring 
the IM progress indicator when conducting IM operations. 

The IM progress indicator used in this experiment did not provide pilots with a 
compelling reason to use it. The spacing error did not help the pilots better under- 
stand the IM commanded speeds, and comments from the post-experiment debrief 
indicated that the progress indicator did not provide pilots with increased under- 
standing of the IM operation. The spacing error that was computed by the spacing 
algorithm used in this experiment depended on several factors, which include the 
target aircraft’s deviation from the published speeds, the IM aircraft’s deviation 
from its published speeds, and the descent wind forecast error. All of these uncer- 
tainties can combine to make the estimated times of arrival unpredictable when the 
aircraft are far from the achieve- by point, and make it difficult for pilots to under- 
stand the reason why a particular IM speed is commanded. Variations within the 
system can cause the spacing error to increase, only to be nulled later in the arrival, 
making it difficult for pilots to extract meaningful information from the indicator. 
Despite these findings, future research could further investigate the results of this 
experiment by examining IM progress indicators during off-nominal operations and 
investigating different IM progress indicator designs. It should also be noted that 
the results from this study may only apply to achieve-by IM operations with the par- 
ticular trajectory-based algorithm that was used. The usefulness of the IM progress 
indicator may change if the operation or spacing algorithm are changed. 

Overall, pilots preferred the graphical AGD over the numerical AGD. Many pilots 
thought that the information presented on the graphical AGD was more intuitive 
than the numerical AGD and that alerting of new IM speed changes was better. 
The graphical AGD was rated as more intuitive and provided pilots with greater 
situational awareness than the numerical AGD. Additionally, the graphical AGD 
contained useful text information, such as the target aircraft and IM mode that were 
not able to be shown on the numerical AGD. Nevertheless, in the post-experiment 
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debrief pilots indicated that the numerical AGD would be sufficient, but not ideal, 
if the saliency of IM speed change alerting was improved. 

5.2 Pilot Performance 

In addition to the survey ratings provided by pilots, two pilot performance metrics 
were examined: the pilots’ reaction time to new commanded speeds and how well 
pilots followed the IM commanded speeds. These metrics add supporting data to 
the workload and acceptability ratings provided by the pilots. 

The hrst metric that was examined was the time it took pilots to recognize a new 
IM speed and begin dialing the speed into the MCP speed window (hgure 17). 
The reaction time is, in essence, a time delay in the control loop. An Analysis 
of Variance (ANOVA) was used to examine the pilots’ reaction time to IM speed 
changes. Since an ANOVA assumes that the data is normally distributed, a log- 
arithm transformation was used to transform the reaction time distributions into 
normal distributions prior to conducting the ANOVA. The results indicate that pi- 
lots had signihcantly faster reaction times when using the graphical AGD [M—11.6, 
SD—9.3, N—263), than when using the numerical AGD [M—18.6, SD—16.1, N—270) 
{p<0.0001). This is likely due to the additional saliency of the speed change alert on 
the graphical AGD. The graphical AGD used reverse video to alert pilots of IM speed 
changes, whereas the numerical AGD used an LED indicator that was signihcantly 
less noticeable. Pilot comments conhrmed that they had difficulty noticing new IM 
speed commands when using the numerical AGD; however, pilot comments also sug- 
gested that the saliency of speed change alerting on the graphical AGD should be 
improved. 

The second pilot performance metric that was examined was the aircraft’s speed 
deviation from the IM speed. In this study, pilots were asked to use the fast /slow 
indicator on the AGD to remain within ten knots of their commanded speed. The 
root-mean-square of the aircraft’s speed deviation from the commanded speed was 
examined to determine if the fast / slow indicator on the graphical AGD helped pilots 
follow the commanded speed. The IM aircraft did have a higher speed deviation when 


Graphical AGD Numerical AGD 



Pilot Reaction Time (sec) Pilot Reaction Time (sec) 

Figure 17: Pilot reaction time to new IM commanded speeds 
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using the numerical AGD {M—6.9, SD—3.2, N—32) than when using the graphical 
AGD {M—5.6, SD—3.0, N—32)] however, the difference was not statistically sig- 
nificant {p—0.08). Furthermore, the difference in the IM aircraft’s speed deviation 
between the graphical AGD and numerical AGD could have been caused by the 
differences in pilot reaction time, in which case the difference may disappear if the 
numerical AGD uses more salient speed change alerting in the future. 

5.3 Spacing Algorithm Performance 

Two measures of spacing algorithm performance were examined in this study: the 
spacing accuracy to the achieve- by point, and the rate at which IM speed changes 
were provided to pilots. The spacing accuracy at the achieve- by point describes how 
accurately and precisely the IM algorithm achieved the assigned spacing goal at the 
achieve-by point (table 5). The spacing accuracy was computed as the difference 
in time between when the target aircraft and IM aircraft crossed the final approach 
fix, subtracted from the assigned spacing interval. Within this experiment, the 
scenarios were scripted so that the assigned spacing goal was always reachable. As 
such, the spacing algorithm was expected to be capable of accurately and precisely 
achieving the spacing goal at the achieve-by point, and the data confirmed these 
expectations. The mean delivery accuracy was within five seconds {p<0.0001), and 
the standard deviation of the delivery accuracy was less than five seconds {p— 0.001). 
Furthermore, all of the IM aircraft arrived within ten seconds of their spacing goal. 
These results are comparable to results from previous research [10,14,15,17,20]. 
An ANOVA revealed a statistically significant difference between the nominal and 
perturbation cases {p— 0.001)] however, the difference is small and is not likely to 
have a large operational impact. 

The rate at which speed changes were provided was an indication of pilot task load 
in this experiment, since the pilots were required to close the control loop by entering 
the commanded speeds into the MGP speed window. The speed change rate was 
computed for three different flight segments: the Genter airspace, within the TRA- 
GON (not including final approach), and on final approach (figure 18). The speed 
change rates observed during this experiment increased as the aircraft approached 
the runway; however, they were always less than two speed changes per minute, 
which has been found to be acceptable in previous studies [5,6]. The pilots were also 
asked to rate the acceptability of the frequency of speed changes using a seven point 
scale ranging from one (completely disagree) to seven (completely agree). On aver- 
age, the pilots moderately agreed that the speed change rate was acceptable {M—6.2, 
SD—1.2, N—128), and moderately agreed that the IM commanded speeds were op- 


Table 5: Delivery accuracy and precision at the final approach fix 



Numerical AGD 

Graphical AGD 


Mean (sec) 

SD (sec) 

Mean (sec) 

SD (sec) 

Nominal 

-2.1 

1.8 

-2.0 

1.6 

Perturbation 

0.1 

2.1 

0.2 

4.1 
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Center TRACON Final Approach 



Speed Changes Per Minute Speed Changes Per Minute Speed Changes Per Minute 

Figure 18: Histograms of IM speed change rate 


erationally acceptable and appropriate {M—6.1, SD=1.3, N—128). Pilot comments 
associated with the individual low ratings indicated speed command frequency, speed 
commands requiring configuration changes, and difficulty noticing speed commands 
as contributing factors. 


6 Conclusions 

This experiment investigated the acceptability and usability of retrofit IM displays 
while conducting nominal IM operations. Two different EFB interfaces and two 
different primary field-of-view AGDs were investigated. Overall, pilot ratings indi- 
cated that the pilot workload and time it took to enter information into both EFB 
interfaces was acceptable. However, pilot comments indicated that the amount of 
data entry required was high and that the operation would be more acceptable if 
the amount of data entry was reduced. A majority of pilots rated both the menu- 
entry and multi-entry EFB interfaces as acceptable; however, pilot comments during 
the post-experiment debrief indicated that pilots had a strong preference for the 
menu-entry EFB interface. 

The graphical AGD was found to be more acceptable and intuitive than the numer- 
ical AGD, with pilot comments indicating that the main reason for the difference in 
acceptability was the alerting saliency of newly generated IM commanded speeds. 
Additional pilot comments indicated that they found the graphical trend indicators 
displayed on the graphical AGD more intuitive than the information displayed on 
the numerical AGD, but pilots only rated the information conveyed by the graph- 
ical indicators as moderately useful. Additionally, pilots ratings indicated that the 
additional text information shown on the graphical AGD was very useful. Based on 
the results from this study, future auxiliary IM displays should have the capability 
of displaying text information such as the target aircraft’s ID and the operational 
state of the IM equipment in the forward field-of-view. 
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Appendix A: Post-Scenario Survey 


I-SIM 

Human-in-the-Loop (HITL) Simulation Study 
POST-RUN QUESTIONNAIRE 

This questionnaire is intended to capture measures of workload and ratings for the events that occurred 
in the scenario that was just completed. You will be asked to complete a more extensive questionnaire at 
the end of the experiment, so please try and keep written comments as concise as possible. 

Administrative Questions 

1. Please circle the scenario you just completed from the list helow: 

• Scenario 1 

• Scenario 2 

• Scenario 3 

• Scenario 4 

• Scenario 5 

• Scenario 6 

• Scenario 7 

• Scenario 8 

2. Please circle your role during the scenario you just completed from the list below: 

• Pilot Flying 

• Pilot Not Flying / Pilot Monitoring 
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Data Entry Phase of Flight 

Average Workload Ratings (Modified Cooper-Harper) 



Difficulty Level 

Operator Demand Level 

Very Easy, 
Highly Desirable 

Operator Mental Effort Is Minimal And 
Desired Performance Is Easily Attainable 

Easy, 

Desirable 

Operator Mental Effort Is Low 

And Desired Performance Is Attainable 

Fair, 

Mild Difficulty 

Acceptable Operator Mental Effort Is 
Required To Attain Adequate System 
Perform ance 


iMInor But Annoying 
iDifficulty 

Moderately High Operator Mental Effort Is 
Required To Attain Adequate System 
Perform ance 


Rating 



( Moderately Objectionable 
Difficulty 


( Very Objectionable But 
Tolerable Difficulty 


Even Though 
Errors May Be 
Large Or Frequent, 
Can Instructed Task 
BeAccomplished 
'“Most Of The Time? 


I Impossible 


High Operator Mental Effort Is Required To 
Attain Adequate System Performance 


MaximumOperator Mental Effort Is Required 
To Attain Adequate System Performance 




Major Difficulty 

Maximum Operator Mental Effort Is Required 
To Bring Errors to Moderate Level 




Major Difficulty 

Maximum Operator Mental Effort Is Required 
To Avoid Large or Numerous Errors 


Major Difficulty 

Intense Operator Mental Effort Is Required 
To Accomplish Task, But Frequent Or 
Numerous Errors Persist 



Instructed Task Cannot 
Be Accomplished Reliably 


Operator Decisions 


3. Follow the flow chart above to select the average workload you experienced during the scenario you 
just completed. 

> Rating of your average workload level while entering information into the EFB: 


32 









4. Respond to each of the statements shown below using a scale ranging from “1” (Completely 
Disagree) to “7” (Completely Agree). Circle one number in conjunction with each statement. 



Rating Scale 


1 2 

3 4 5 

6 7 

Completely 


Completely 

Disagree 


Agree 


Relevant information, including operational 
plans, decisions, and changes in aircraft state 
were effectively communicated between 
yourself and your crewmember during the 
data entry phase of flight. 

1 2 3 4 5 6 7 

The time available for tasks during the data 
entry phase of flight was well managed. 

1 2 3 4 5 6 7 

The use of voice communications to provide 
the rM clearance was acceptable in this 
scenario. 

1 2 3 4 5 6 7 

The amount of head down time required to 
input information from the IM clearance into 
the EFB was acceptable. 

1 2 3 4 5 6 7 

During this scenario, entering IM clearance 
information into the EFB was easy and 
intuitive. 

1 2 3 4 5 6 7 

The flight crew procedures for receiving and 
entering an IM clearance were complete and 
acceptable. 

1 2 3 4 5 6 7 

Entering information into the EFB resulted in 
a minimal amount of frustration. 

1 2 3 4 5 6 7 


5. Please briefly explain any undesirable ratings from the statements above: 
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Monitoring IM Phase of Flight 

Average Workload Ratings (Modified Cooper-Harper) 


Difficulty Level Operator Demand Level Rating 



Even Though 
Errors May Be 
Large Or Frequent. 
Can Instructed Task 
BeAccomplished 
''Most Of The Time? 



Very Easy, 
Highiy Desirable 

Operator Mental Effort Is Minimal And 
Desired Performance Is Easily Attainable 

Easy, 

Operator Mental Effort Is Low 

Desirable 

And Desired Performance Is Attainable 

Fair, 

Acceptable Operator Mental Effort Is 

Mild Difficulty 

Required To Attain Adequate System 


Performance 



Minor But Annoying 
Difficulty 

Moderately High Operator Mental Effort Is 
Required To Attain Adequate System 
Performance 



Moderately Objectionable 
Difficulty 

High Operator Mental Effort Is Required To 
Attain Adequate System Performance 

■1 

Very Objectionable But 
Tolerable Difficulty 

MaximumOperator Mental Effort Is Required 
To Attain Adequate System Performance 


Major 

Deficiencies 
System 
Redesign Is 
Strongly 
Recommended 




Major Difficulty 

Maximum Operator Mental Effort Is Required 
To Bring Errors to Moderate Level 




Major Difficulty 

Maximum Operator Mental Effort Is Required 
To Avoid Large or Numerous Errors 

■1 

Major Difficulty 

Intense Operator Mental Effort Is Required 
To Accomplish Task, But Frequent Or 
Numerous Errors Persist 





Instructed Task Cannot 
Be Accomplished Reliably 



Operator Decisions 


6. Follow the flow chart above to select the average workload you experienced during the scenario you 
just completed. 

> Rating of your average workload level while monitoring the FIM operation (when 
speed guidance was being provided) 
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7. Respond to each of the statements shown below using a scale ranging from “1” (Completely 
Disagree) to “7” (Completely Agree). Circle one number in conjunction with each statement. 



Rating Scale 


1 2 

3 4 5 

6 7 

Completely 


Completely 

Disagree 


Agree 


Relevant information, including operational 
plans, decisions, and changes in aircraft state 
were effectively communicated between 
yourself and your crewmember while 
monitoring the IM operation. 

1 2 3 4 5 6 7 

The time available for tasks was well 
managed while monitoring the IM operation. 

1 2 3 4 5 6 7 

The IM commanded speeds were 
operationally acceptable and appropriate. 

1 2 3 4 5 6 7 

The frequency of the IM speed commands 
was acceptable at all times throughout the 
scenario. 

1 2 3 4 5 6 7 

I understood why the IM commanded speeds 
were provided (i.e. the IM commanded 
speeds made sense). 

1 2 3 4 5 6 7 

The use of voice communications to provide 
the IM clearance(s) was acceptable in this 
scenario. 

1 2 3 4 5 6 7 

The flight crew procedures for the events in 
this scenario were complete and acceptable. 

1 2 3 4 5 6 7 

During this scenario, it was easy to obtain 
needed information from the primary field of 
view IM displays (i.e. the primary field of 
view displays were intuitive). 

1 2 3 4 5 6 7 

The primary field of view displays provided 
me with adequate situational awareness. 

1 2 3 4 5 6 7 


8. Please briefly explain any undesirable ratings from the statements above: 
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All Phases of Flight 

9. Describe any unusual or unexpected event(s) and your reaction(s), if applicable: 


10. (Optional) This space is reserved for any additional comments related to awareness and 

acceptability issues. If you have any clarifying comments or interesting observations related to 
awareness and acceptability issues, please provide them below. 
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Appendix B: Post-Experiment Survey 


I-SIM 

Human-in-the-Loop (HITL) Simulation Study 
POST-EXPERIMENT QUESTIONNAIRE 

Post Experiment Questionnaire 

This questionnaire is intended to gather your comments and suggestions regarding the experiment itself 
as well as Interval Management concept 


This questionnaire contains items associated with each of the following categories: 

• Simulator and Flight Scenarios 

• Training 

• Interval Management Procedures 

• Interval Management Displays 

• Spacing Tool 

• Additional Comments 


Simulator and Flight Scenarios 

1. Was the workload required to operate the simulator much less than, the same as, or greater 
than the workload required to fly an actual aircraft? 


Moderately 

Slightly 

Slightly 

Moderately 

Much More More 

More 

The Same Less 

Less Much Less 

1 2 

3 

4 5 

6 7 


Please provide any additional comments regarding the simulator: 


2. Please share your Impressions of the flight scenarios (e.g., comment on their level of realism, 
appropriateness, and/or diversity) and comment on how the design of the scenarios Impacted 
your ability to perform the spacing task: 
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Training 


3. Did you receive adequate training with respect to flying the simulator? 

YES 

NO 

If not, briefly describe how simulator training can be improved: 


4. Did you receive adequate training with respect to the IM spacing procedure and the spacing 
tool? 


YES 

NO 

If not, briefly describe how IM procedure or spacing tool training can be Improved: 


5. Did you receive adequate training with respect to the entry and interpretation of information 
presented on the EFB interfaces? 

YES 

NO 

If not, briefly describe how EFB training can be Improved: 


6. Did you receive adequate training with respect to the interpretation of Information presented 
on the CGD and AGD? 

YES 

NO 

If not, briefly describe how AGD training can be Improved: 
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Interval Management Procedures 

7. Were the IM procedures complete, accurate, and logical? 

YES 

NO 

Please provide any suggestions regarding the way(s) in which the general IM procedures may 
he improved: 


8. Was the IM phraseology used in this experiment correct and intuitive? 

YES 

NO 

If “no,” why not, and what could he done to improve the phraseology? 


9. How difficult do you think it would he for a typical flight crew to learn and Integrate the IM 
spacing procedures into their current daily operational flight procedures? 


Very 

Moderately 

Slightly 


Slightly 

Moderately 

Very 

Difficult 

Difficult 

Difficult 

Neutral 

Easy 

Easy 

Easy 

1 

2 

3 

4 

5 

6 

7 


Briefly descrihe any challenges Involved with integrating the IM procedures with existing 
procedures: 


10. Do you think the division of tasks between the Pilot Flying (PF) and Pilot Monitoring (PM) 
was hoth desirable and fit within the current distribution of tasks between PF and PM? 

YES 

NO 

If “no,” what was wrong with the division, and how would you reallocate the tasks? 
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11. Given the experience with IM that you gained during this simulation, what is your overall 
assessment of the safety of the spacing procedure compared with current day operations? 
(“Safety” in this question refers to your holistic opinion to include workload, awareness, 
position relative to other aircraft, etc.) 


Not Safe At 

Moderately 

Slightly 

Slightly 

Moderately 

Much More 

All 

Less Safe 

Less Safe 

As Safe More Safe 

More Safe 

Safe 

1 

2 

3 

4 5 

6 

7 


Briefly describe any characteristic or event that determined your rating (if appropriate): 


Interval Management Displays 
EFB Displays 
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Figure 1: Menu-entry EFB interface 
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Figure 2: Multi-entry EFB interface 


12. In general, did you find the process of entering IM clearance information into the menu-entry 
EFB interface easy and intuitive? 

YES 

NO 

Please describe any improvements that could be made to the menu-entry EFB interface. 
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13. Do you think the menu-entry EFB interface is a minimally acceptahle Interface for entering 
IM information (i.e. if the menu-entry EFB Interface was placed on an aircraft, could you use 
it to enter IM information without compromising safety or experiencing too high of a 
workload). 

YES 

NO 

If “no,” why isn’t the menu-entry EFB Interface minimally acceptable? 


14. In general, did you find the process of entering IM clearance Information into the multi-entry 
EFB interface easy and Intuitive? 

YES 

NO 

Please describe any Improvements that could be made to the multi-entry EFB Interface 


15. Do you think the multi-entry EFB interface is a minimally acceptable Interface for entering IM 
information (i.e. if the multi-entry EFB Interface was placed on an aircraft, could you use it to 
enter IM information without compromising safety or experiencing too high of a workload). 

YES 

NO 

If “no,” why isn’t the multi-entry EFB Interface minimally acceptable? 


16. If you were conducting IM operations into a busy terminal area, would you prefer using the 
menu-entry or multi-entry EFB Interface to enter IM information? 

Menu-entry 

Multi-entry 

Please briefly explain your answer. 
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Primary Field of View Displays 
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Figure 3: Configurable Glass Display (CGD) Figure 4: ADS-B Guidance Display (AGD) 


17. Using a scale ranging from “Detrimental (Hurts Performance)” to “Could Not Do IM 

Without”, rate the usefulness of the IM symbology on the Configurable Glass Display (CGD). 



Detrimental 

(Hurts 

Performance) 

Not 
Useful 
At All 

Slightly 

Useful 

Moderately 

Useful 

Very 

Useful 

Required 
for IM 

Fast/slow indicator 

1 

2 

3 

4 

5 

6 

Fast/slow acceleration arrow 

1 

2 

3 

4 

5 

6 

IM commanded speed 

1 

2 

3 

4 

5 

6 

IM mode 

1 

2 

3 

4 

5 

6 

Lead aircraft callsign 

1 

2 

3 

4 

5 

6 

IM progress indicator 

1 

2 

3 

4 

5 

6 

Drag required message 

1 

2 

3 

4 

5 

6 


18. Please explain any undesirable ratings from the statements above. 


19. Do you think the CGD is a minimally acceptable display for monitoring the IM operation (i.e. 
if the CGD was placed on an aircraft, could you use it to effectively monitor the IM operation 
without compromising safety or experiencing too high of a workload). 

YES 

NO 

If “no,” why wouldn’t the CGD be minimally acceptable? 
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20. Did the CGD provide you with the information you needed/desired to safely and correctly 
conduct IM, and was this information easy to obtain when needed? 

YES 

NO 

Please provide any changes you would make to the CGD to Improve it? 


21. Using a scale ranging from “Detrimental (Hurts Performance)” to “Could Not Do IM 
Without”, rate the usefulness of the IM symbology on the ADS-B Guidance AGD. 



Detrimental 

(Hurts 

Performance) 

Not 
Useful 
At All 

Slightly 

Useful 

Moderately 

Useful 

Very 

Useful 

Required 
for IM 

Commanded Speed 

1 

2 

3 

4 

5 

6 

East/Slow Number 

1 

2 

3 

4 

5 

6 

Spacing Error 

1 

2 

3 

4 

5 

6 


22. Please explain any undesirable ratings from the statements above. 


23. Do you think the AGD is a minimally acceptable display for monitoring the IM operation (i.e. 
if the AGD was placed on an aircraft, could you use it to effectively monitor the IM operation 
without compromising safety or experiencing too high of a workload). 

YES 

NO 

If “no,” why wasn’t the AGD be minimally acceptable? 


24. Did the AGD provide you with the Information you needed/desired to safely and correctly 
conduct IM, and was this information easy to obtain when needed? 

YES 

NO 

Please provide any changes you would make to the AGD to Improve it? 
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25. Would you prefer using the AGD or CGD to monitor the IM operation if you were conducting 
IM operations into a busy terminal area? 

AGD 

CGD 

Please briefly explain your answer. 


Function Allocation and the Spacing Tool 

26. Did following the IM commanded speed and procedure ever cause unexpected or undesirable 
behavior? 

YES 

NO 

If “yes,” please explain what the unexpected or undesirable behavior was: 


27. Did you find the responsibility of using onboard automation to achieve a spacing interval 
behind a lead aircraft acceptable (when ATC is responsible for separation)? 

YES 

NO 

If “no,” why not, and what could be done to make the responsibility or workload acceptable? 


28. Did you find your level of engagement with the IM automation acceptable (i.e. the level of 
decision making ability you had, and your understanding of the reasoning behind IM speeds 
that were commanded)? 

YES 

NO 

Please explain your answer. 


Additional Comments 

29. Do you have any additional comments about the experiment? 
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